archived/deep_demand_forecasting/deep_demand_forecast.ipynb (3,203 lines of code) (raw):

{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Deep Demand Forecasting with Amazon SageMaker" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "---\n", "\n", "This notebook's CI test result for us-west-2 is as follows. CI test results in other regions can be found at the end of the notebook. \n", "\n", "![This us-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-2/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "---" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**Jupyter Kernel**:\n", "* Please make sure you are using **Python 3 (MXNet 1.9 Python 3.8 CPU Optimized)** kernel\n", "\n", "**Run All**: \n", "\n", "* If you are in SageMaker Notebook instance, you can *go to Cell tab -> Run All*\n", "* If you are in SageMaker Studio, you can *go to Run tab -> Run All Cells*\n", "\n", "**Overview**\n", "\n", "The deep demand forecasting solution contains five sections.\n", "\n", "* Stage I: Data preparation and visualization.\n", "* Stage II: Train an optimal LSTNet model using GluonTS with Hyper-Parameter Optimization (HPO).\n", "* Stage III: Train an optimal SageMaker DeepAR model with HPO.\n", "* Stage IV: Evaluate model performance of all three algorithms on the same hold-out test data." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" }, "tags": [] }, "outputs": [], "source": [ "# Install dependencies files that are used in this notebook.\n", "\n", "!pip install altair lunarcalendar plotly\n", "!pip install gluonts==0.8.1\n", "!pip install pystan==2.19.1.1" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import warnings\n", "import json\n", "import sagemaker\n", "\n", "warnings.filterwarnings(\"ignore\")\n", "session = sagemaker.Session()\n", "\n", "role = sagemaker.get_execution_role()\n", "solution_bucket = \"sagemaker-solutions-prod-us-west-2\"\n", "solution_name = \"Deep-demand-forecasting-with-amazon-sagemaker/2.0.1\"\n", "bucket = session.default_bucket()\n", "training_instance_type = \"ml.p3.2xlarge\"\n", "inference_instance_type = \"ml.g4dn.2xlarge\"\n", "solution_prefix = \"sagemaker-soln-ddf-js--\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stage I: Data Preparation and Visualization\n", "\n", "### Copy Raw Data to S3\n", "\n", "The dataset we use here is the multivariate time-series [electricity consumptions](https://archive.ics.uci.edu/ml/datasets/ElectricityLoadDiagrams20112014) data taken from Dua, D. and Graff, C. (2019). [UCI Machine Learning Repository](http://archive.ics.uci.edu/ml/index.php), Irvine, CA: University of California, School of Information and Computer Science. A cleaned version of the data containing **321** time-series with **1 Hour** frequency, starting from **2012-01-01** with **26304** time-steps, is available to download directly via [GluonTS](https://gluon-ts.mxnet.io/). We have also provided the [exchange rate](https://github.com/laiguokun/multivariate-time-series-data/tree/master/exchange_rate) dataset in case you want to try with other datasets as well.\n", "\n", "For the ease of access, with have made both of the cleaned datasets available in the following S3 bucket.\n", "\n", "**Note:** To reproduce the results from the [blog post](https://towardsdatascience.com/deep-demand-forecasting-with-amazon-sagemaker-e0226410763a) we use `DATASET_NAME=\"electricity\"`." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "DATASET_NAME = \"electricity\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.s3 import S3Downloader\n", "\n", "original_bucket = f\"s3://{solution_bucket}/0.2.0/{solution_name}\"\n", "original_data_prefix = f\"artifacts/data/{DATASET_NAME}\"\n", "original_data = f\"{original_bucket}/{original_data_prefix}\"\n", "print(\"original data: \")\n", "S3Downloader.list(original_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "First, setup the S3 bucket name and prefix" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "prefix = \"tst\" # example\n", "raw_data = f\"s3://{bucket}/{prefix}\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Copy the `original_data` (s3 bucket in the source account) to our `raw_data` (s3 bucket in your account) if does not exist already" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "\n", "if not S3Downloader.list(raw_data):\n", " !aws s3 cp --recursive $original_data $raw_data" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Download the dependencies wheel files from source S3 to local directory for the usage of `training` and `inference` containers." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "original_dependencies_bucket = f\"{original_bucket}/artifacts/dependencies\"\n", "\n", "# Download the dependencies wheel files from source S3 to local directory for the usage of training and inference containers\n", "!aws s3 cp --recursive --no-progress $original_dependencies_bucket lstnet/dependencies" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Set a few variables that are used throughout the notebook" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "input_data = raw_data\n", "train_output = f\"s3://{bucket}/{prefix}/output\"\n", "code_location = f\"s3://{bucket}/{prefix}/code\"" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualize Data\n", "\n", "We have provided some utilities in `lstnet/monitor.py` for creating the dataframe from train and test data.\n", "\n", "The multivariate time-series electricity consumptions data contains **321** time-series (households) \n", "with **1 Hour** frequency ranging from year **2012** to **2014**. The training data include hourly electricity consumption values (for the 321 households) \n", "from **2012-01-01 00:00:00** to **2014-05-26 19:00:00**, and the test data contains values from **2012-01-01 00:00:00** to **2014-06-02 19:00:00** (7 more days of hourly data compared to the training data). " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "#### Define **CONTEXT_LENGTH** and **PREDICTION_LENGTH**\n", "To train a time series forecasting model, the **CONTEXT_LENGTH** defines the length of each input time series, and **PREDICTION_LENGTH** defines the length of each output time series." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "CONTEXT_LENGTH = 168 # input length corresponding to 7 days\n", "PREDICTION_LENGTH = 24 # output length corresponding to 1 day" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "\n", "As the `CONTEXT_LENGTH` and `PREDICTION_LENGTH` are set to 168 (7 days) and 24 (next 1 day), we plot the last 7 days of the training data and its subsequent 1 day of the testing data for demonstration purpose. \n", "The plotted training data and testing data are from **2014-05-19 20:00:00** to **2014-05-26 19:00:00**, and from **2014-05-26 20:00:00** to **2014-05-27 02:00:00**, respectively. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Prepare the data for visualization." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "!mkdir -p raw_data\n", "!aws s3 cp --recursive $original_data raw_data\n", "%run lstnet/monitor.py\n", "\n", "train_df, test_df = prepare_data(\"raw_data\")\n", "NUM_TS = train_df.shape[1] - 1\n", "print(f\"raw train data shape {train_df.shape}, test data shape {test_df.shape}\")\n", "ts_col_names = [f\"ts_{i}\" for i in range(NUM_TS + 1)]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_df_viz, test_df_viz, selected_cols = create_data_viz(\n", " train_df, test_df, CONTEXT_LENGTH, PREDICTION_LENGTH, num_sample_ts=11\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "tags": [] }, "outputs": [], "source": [ "!pip install typing_extensions==4.4.0\n", "import typing_extensions\n", "from importlib import reload\n", "reload(typing_extensions)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "For demonstration purpose, we only plot the first 11 time series out of the 321 time series." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import altair as alt\n", "from pathlib import Path\n", "from gluonts.dataset.common import MetaData\n", "\n", "\n", "selection = alt.selection_multi(fields=[\"covariate\"], bind=\"legend\", nearest=True)\n", "\n", "train_plot = (\n", " alt.Chart(train_df_viz, title=\"Input data\")\n", " .mark_line()\n", " .encode(\n", " alt.X(\"time:T\", axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\")),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 1300]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "meta = MetaData.parse_file(Path(\"raw_data\") / \"metadata.json\")\n", "timestamps_test = pd.date_range(test_df_viz.iloc[0][\"time\"], periods=CONTEXT_LENGTH, freq=meta.freq)\n", "ts_start, ts_end = timestamps_test[0], timestamps_test[-1]\n", "test_plot = (\n", " alt.Chart(test_df_viz, title=\"Expected output data (ground truth)\")\n", " .mark_line()\n", " .encode(\n", " alt.X(\n", " \"time:T\",\n", " axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\"),\n", " scale=alt.Scale(domain=[ts_start, ts_end]),\n", " ),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 1300]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "train_plot | test_plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stage II: Train an Optimal LSTNet Model using GluonTS with Hyper-Parameter Optimization (HPO)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "**LSTNet** is a Deep Learning model that incorporates traditional auto-regressive linear models in parallel to the non-linear neural network part, which makes the non-linear deep learning model more robust for the time series which violate scale changes.\n", "\n", "For information on the mathematics behind LSTNet, see [Modeling Long- and Short-Term Temporal Patterns with Deep Neural Networks](https://arxiv.org/abs/1703.07015).\n", "\n", "We summarize key actions in this stage as below.\n", "\n", "* We firstly train a LSTNet model without HPO. \n", "* Next, we train an optimal LSTNet model with HPO. \n", "* We demonstrate that the LSTNet trained with HPO significantly outperform the one trained without HPO." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Hyperparameters\n", "\n", "Here is a set of hyperparameters for LSTNet model." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hyperparameters = {\n", " \"context_length\": CONTEXT_LENGTH, # sliding window size for training\n", " \"prediction_length\": PREDICTION_LENGTH, # sliding window size for predicting\n", " \"skip_size\": 4, # skip size is used in skip-rnn layer.\n", " \"ar_window\": 4, # auto-regressive window size\n", " \"channels\": 72, # number of convolution channels for the first layer. Note. channels should be divisible by skip_size\n", " \"scaling\": True, # whether to scale the data or not\n", " \"output_activation\": \"sigmoid\", # output activation function either None, sigmoid or tanh\n", " \"epochs\": 15, # number of epochs for training\n", " \"batch_size\": 32, # number of batch size\n", " \"learning_rate\": 1e-2, # learning rate for weight update\n", " \"dropout_rate\": 0.2, # dropout regularization parameter\n", " \"rnn_cell_type\": \"gru\", # type of the RNN cell. Either lstm or gru\n", " \"rnn_num_layers\": 1, # number of RNN layers to be used\n", " \"rnn_num_cells\": 100, # number of RNN cells for each layer\n", " \"skip_rnn_cell_type\": \"gru\", # type of the RNN cell for the skip layer. Either lstm or gru\n", " \"skip_rnn_num_layers\": 1, # number of RNN layers to be used for skip part\n", " \"skip_rnn_num_cells\": 10, # number of RNN cells for each layer for skip part\n", " \"lead_time\": 0, # lead time\n", " \"kernel_size\": 6, # kernel size for first layer Conv2D\n", "}" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Train a LSTNet Model without HPO using SageMaker Estimator\n", "\n", "With the hyperparameters defined, we can execute the training job. We use the [GluonTS](https://gluon-ts.mxnet.io/), with **MXNet** as the backend deep learning framework, to define and train our LSTNet model. **Amazon SageMaker** makes it do this with the Framework estimators, which have the deep learning frameworks already setup. Here, we create a SageMaker MXNet estimator and pass in our model training script, hyperparameters, as well as the number and type of training instances we want.\n", "\n", "We can then `fit` the estimator on the the `input_data` location in S3. Specifically, the `input_data` contains training and test data, which are in `input_data/train/data.json` and `input_data/test/data.json`, respectively. The training data contains 321 time series from from **2012-01-01 00:00:00** to **2014-05-26 19:00:00**, and the test data contains the same time series from **2012-01-01 00:00:00** to **2014-06-02 19:00:00** (7 more days of hourly data compared to the training data). \n", "\n", "\n", "The first 6 days of the test data are further considered as validation data. When HPO is enabled, the optimal hyperparameters are selected based on the validation data. The last 1 day of the test data is considered as the hold-out test set to evaluate the model performance.\n", "\n", "\n", "Note. When a trained model is evaluated on the test set, only the trailing **PREDICTION_LENGTH** (24) observations of each time series are predicted. The reason to keep entire time series as input (which starts from **2012-01-01 00:00:00**) is to ensure there are enough input observations (i.e., **CONTEXT_LENGTH** observations) for model to make predictions for the trailing **PREDICTION_LENGTH** observations. The evaluation function `evaluate` is used in `lstnet/train.py` and defined in `lstnet/utils.py`." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Training the estimator for 15 epochs takes around **10 minutes**." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "import time\n", "from sagemaker.mxnet import MXNet\n", "import uuid\n", "\n", "!cp lstnet/requirements_train.txt lstnet/requirements.txt\n", "unique_hash = str(uuid.uuid4())[:6]\n", "training_job_name = solution_prefix + unique_hash + \"-lstnet-training\"\n", "print(\n", " f\"You can go to SageMaker -> Training -> Training jobs -> a job name started with {training_job_name} to monitor training status and details.\"\n", ")\n", "\n", "estimator = MXNet(\n", " entry_point=\"train.py\",\n", " source_dir=\"lstnet\",\n", " role=role,\n", " instance_count=1,\n", " instance_type=training_instance_type, # or \"ml.m5.4xlarge\" without GPU\n", " framework_version=\"1.8.0\",\n", " py_version=\"py37\",\n", " hyperparameters=hyperparameters,\n", " base_job_name=training_job_name,\n", " output_path=train_output,\n", " code_location=code_location,\n", " sagemaker_session=session,\n", " enable_network_isolation=True, # Set enable_network_isolation=True to ensure a security running environment\n", " # container_log_level=logging.DEBUG, # display debug logs\n", " env={\"MMS_DEFAULT_RESPONSE_TIMEOUT\": \"1000\"},\n", ")\n", "\n", "start_time = time.time()\n", "\n", "estimator.fit(input_data, logs=False)\n", "\n", "single_training_job_time_duration = time.time() - start_time # unit: minute" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Examine the Training Evaluation\n", "\n", "We can now access the training artifacts from the specified `output_path` in the above estimator and visualize the training results. \n", "\n", "The current training output are saved in `output.tar.gz` format and donwloaded into notebook for visulization." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import os\n", "\n", "output_path = os.path.join(train_output, estimator._current_job_name, \"output\")\n", "\n", "S3Downloader.download(output_path, \"output\")\n", "!tar -xvf output/output.tar.gz -C output/" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import pandas as pd\n", "\n", "item_metrics = pd.read_csv(\"output/item_metrics.csv.gz\", compression=\"gzip\")\n", "item_metrics.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualizing the Training Evaluation\n", "\n", "For the visualization we use [altair package](https://github.com/altair-viz/altair) with declarative API. If you want to export to different file formats, follow [altair_saver](https://github.com/altair-viz/altair_saver). \n", "\n", "Note that after exporting to `html` you can go to `output` and open the generated `html` files inside notebook.\n", "\n", "Here, we compare the [**Mean Absolute Scaled Error (MASE)**](https://en.wikipedia.org/wiki/Mean_absolute_scaled_error) against the [**symmetric Mean Absolute Percentage Error (sMAPE)**](https://en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error). For both metrics, smaller values indicate bettter performance. Thus, the closer the scatter points come to the lower left corner in the plot below, the more accurate the predictions are." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "col_a = \"MASE\"\n", "col_b = \"sMAPE\"\n", "\n", "alt.data_transformers.disable_max_rows()\n", "\n", "scatter = (\n", " alt.Chart(item_metrics)\n", " .mark_circle(size=100, fillOpacity=0.8)\n", " .encode(\n", " alt.X(col_a, scale=alt.Scale(domain=[-0.5, 10])),\n", " alt.Y(col_b, scale=alt.Scale(domain=[0, 2.5])),\n", " tooltip=[col_a, col_b],\n", " )\n", " .interactive()\n", ")\n", "scatter.save(os.path.join(\"output\", f\"{col_a}_vs_{col_b}.html\"))\n", "scatter" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "col_a_plot = (\n", " alt.Chart(item_metrics)\n", " .mark_bar()\n", " .encode(\n", " alt.X(col_a, bin=True),\n", " y=\"count()\",\n", " )\n", ")\n", "col_b_plot = (\n", " alt.Chart(item_metrics)\n", " .mark_bar()\n", " .encode(\n", " alt.X(col_b, bin=True),\n", " y=\"count()\",\n", " )\n", ")\n", "\n", "col_a_b_plot = col_a_plot | col_b_plot\n", "col_a_b_plot.save(os.path.join(\"output\", f\"{col_a}_{col_b}_barplots.html\"))\n", "col_a_b_plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deploy an Endpoint\n", "\n", "To serve the model, we can deploy an endpoint (which takes around **7 minutes**) where the `lstnet/inference.py` script handles the predictions using the trained model as follows." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "from sagemaker.mxnet import MXNetModel\n", "\n", "!cp lstnet/requirements_inference.txt lstnet/requirements.txt\n", "\n", "endpoint_name = solution_prefix + unique_hash + \"-lstnet-endpoint\"\n", "print(\n", " f\"You can go to SageMaker -> Inference -> Endpoints --> an endpoint with name {endpoint_name} to monitor the deployment status.\"\n", ")\n", "\n", "model = MXNetModel(\n", " model_data=os.path.join(output_path, \"model.tar.gz\"),\n", " role=role,\n", " entry_point=\"inference.py\",\n", " source_dir=\"lstnet\",\n", " py_version=\"py37\",\n", " name=solution_prefix + \"-model\",\n", " framework_version=\"1.8.0\",\n", " code_location=code_location,\n", " enable_network_isolation=True, # Set enable_network_isolation=True to ensure a security running environment\n", " env={\"MMS_DEFAULT_RESPONSE_TIMEOUT\": \"1000\"},\n", ")\n", "\n", "predictor = model.deploy(\n", " instance_type=inference_instance_type,\n", " endpoint_name=endpoint_name,\n", " initial_instance_count=1,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Testing the Endpoint\n", "\n", "To do sanity checking, here we can test the endpoint by requesting predictions for a randomly generated data. The `predictor` handles serialization and deserialization of the requests." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_df, test_df = prepare_data(\"raw_data\")\n", "NUM_TS = train_df.shape[1] - 1" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "import numpy as np\n", "\n", "np.random.seed(1)\n", "random_test = np.random.randn(NUM_TS, PREDICTION_LENGTH)\n", "\n", "# json serializable request format\n", "random_test_data = {}\n", "random_test_data[\"target\"] = random_test.tolist()\n", "random_test_data[\"start\"] = \"2014-01-01\"\n", "random_test_data[\"source\"] = []\n", "\n", "random_ret = predictor.predict(random_test_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Load the returned JSON objects into numpy array." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "forecasts = np.array(random_ret[\"forecasts\"][\"samples\"])\n", "print(f\"Forecasts shape with 10 samples: {forecasts.shape}\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Query the Endpoint for Prediction and Compute the Evaluation Metrics\n", "Here we prepare our test data for prediction. Since we have trained our model given the hyperparameters defined earlier `CONTEXT_LENGTH` (length of input time series) and `PREDICTION_LENGTH` (length of output time series), \n", "we can now input the final window in the test data (a hold-out data without being used in the training and validation) to our model to test from **2014-06-01 20:00:00** onwards, get the predictions, and compute the evaluation metrics." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "%%time\n", "\n", "start_time = time.time()\n", "test_data = {}\n", "test_data[\"target\"] = (\n", " test_df.iloc[-(CONTEXT_LENGTH + PREDICTION_LENGTH) :].set_index(\"time\").values.T.tolist()\n", ")\n", "test_data[\"start\"] = \"2014-05-25 20:00:00\"\n", "test_data[\"source\"] = []\n", "\n", "predictions = predictor.predict(test_data)\n", "inference_time_duration = time.time() - start_time" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Summarize the evaluation metrics on the test data" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "dict_agg_metrics_without_hpo = json.loads(predictions[\"agg_metrics\"])\n", "dict_agg_metrics_without_hpo[\"RRSE\"] = dict_agg_metrics_without_hpo[\"RMSE\"] / np.std(\n", " test_data[\"target\"]\n", ") # compute Root Relative Squared Error (RRSE)\n", "dict_agg_metrics_without_hpo[\"Training Time (min)\"] = single_training_job_time_duration / 60\n", "dict_agg_metrics_without_hpo[\"Inference Time (s)\"] = inference_time_duration\n", "df_agg_metrics_without_hpo = pd.DataFrame.from_dict(dict_agg_metrics_without_hpo, orient=\"index\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Compute the Evaluation Metrics\n", "Except for the training and inference time, for RRSE (Root Relative Squared Error), MAPE (Mean Absolute Percentage Error), and sMAPE (symmetric Mean Absolute Percentage Error), smaller values indicate better predictive performance." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "METRICS_TO_SHOW = [\"RRSE\", \"MAPE\", \"sMAPE\", \"Training Time (min)\", \"Inference Time (s)\"]\n", "print(\"Evaluation metrics on the test data are shown as below.\")\n", "df_agg_metrics_without_hpo_eval_metrics = df_agg_metrics_without_hpo.reindex(METRICS_TO_SHOW)\n", "df_agg_metrics_without_hpo_eval_metrics.columns = [\"LSTNet without HPO\"]\n", "print(\"For the evaluation metrics, smaller value indicates better predictive performance.\")\n", "df_agg_metrics_without_hpo_eval_metrics" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Interactive Visualization\n", "\n", "It is important to visualize how the model is performing given the test data. Based on the predictions from **2014-06-01 20:00:00** onwards by querying the endpoint shown above, \n", "we visualize the performance of the model for a sample of time-series." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f\"prepared train data shape {train_df.shape}, test data shape {test_df.shape}\")\n", "ts_col_names = [f\"ts_{i}\" for i in range(NUM_TS + 1)]\n", "train_df_viz, test_df_viz, selected_cols = create_data_viz(\n", " test_df.iloc[: test_df.shape[0] - PREDICTION_LENGTH],\n", " test_df,\n", " CONTEXT_LENGTH,\n", " PREDICTION_LENGTH,\n", " num_sample_ts=test_df.shape[1] - 1,\n", ")\n", "train_df_viz.head()" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from gluonts.dataset.common import ListDataset\n", "from gluonts.dataset.field_names import FieldName\n", "\n", "forecasts = np.transpose(np.array(predictions[\"forecasts\"][\"samples\"][0]))\n", "\n", "preds = ListDataset(\n", " [\n", " {\n", " FieldName.TARGET: forecasts,\n", " FieldName.START: predictions[\"forecasts\"][\"start_date\"],\n", " }\n", " ],\n", " freq=predictions[\"forecasts\"][\"freq\"],\n", " one_dim_target=False,\n", ")\n", "\n", "preds_df = multivar_df(next(iter(preds)))\n", "preds_df_filter = preds_df.loc[:, [\"time\"] + selected_cols]\n", "preds_df_filter = pd.melt(preds_df_filter, id_vars=[\"time\"], value_vars=selected_cols)\n", "preds_df_filter.rename(columns={\"variable\": \"covariate\"}, inplace=True)\n", "preds_df_filter.head()" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "For demonstration purpose, we randomly select 10 time series to visualize model predictions to compare with the ground truth data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "selected_ts = [\n", " \"ts_98\",\n", " \"ts_100\",\n", " \"ts_116\",\n", " \"ts_137\",\n", " \"ts_147\",\n", " \"ts_156\",\n", " \"ts_184\",\n", " \"ts_216\",\n", " \"ts_245\",\n", " \"ts_267\",\n", "]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "selection = alt.selection_multi(fields=[\"covariate\"], bind=\"legend\", nearest=True)\n", "\n", "train_plot = (\n", " alt.Chart(\n", " train_df_viz.loc[train_df_viz[\"covariate\"].isin(selected_ts)],\n", " title=\"Input data\",\n", " )\n", " .mark_line()\n", " .encode(\n", " alt.X(\"time:T\", axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\")),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 3000]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "timestamps_test = pd.date_range(test_df_viz.iloc[0][\"time\"], periods=CONTEXT_LENGTH, freq=meta.freq)\n", "ts_start, ts_end = timestamps_test[0], timestamps_test[-1]\n", "test_plot = (\n", " alt.Chart(\n", " test_df_viz.loc[test_df_viz[\"covariate\"].isin(selected_ts)],\n", " title=\"Expected output data (ground truth)\",\n", " )\n", " .mark_line()\n", " .encode(\n", " alt.X(\n", " \"time:T\",\n", " axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\"),\n", " scale=alt.Scale(domain=[ts_start, ts_end]),\n", " ),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 3000]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "preds_plot = (\n", " alt.Chart(\n", " preds_df_filter.loc[preds_df_filter[\"covariate\"].isin(selected_ts)],\n", " title=\"Model predictions\",\n", " )\n", " .mark_line()\n", " .encode(\n", " alt.X(\n", " \"time:T\",\n", " axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\"),\n", " scale=alt.Scale(domain=[ts_start, ts_end]),\n", " ),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 3000]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "(train_plot | test_plot) & preds_plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Note. The model predictions shown in above plots are not optimal as the HPO is not enabled. In next section, we train an optimal LSTNet model using HPO. The visualizations of model predictions from the optimal LSTNet are shown in the Stage V with those from all the other models." ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "### Train an Optimal LSTNet Model with HPO using SageMaker Estimator\n", "\n", "In this section we further improve the model performance by adding HPO tuning with [SageMaker Automatic Model Tuning](https://docs.aws.amazon.com/sagemaker/latest/dg/automatic-model-tuning.html). \n", "Amazon SageMaker automatic model tuning, also known as hyperparameter tuning, finds the best version of a model by running many training jobs on your dataset using the algorithm and ranges of hyperparameters that you specify. It then chooses the hyperparameter values that result in a model that performs the best, as measured by a metric that you choose.\n", "The best model and its corresponded hyperparmeters are selected on the validation data from **2014-05-26 20:00:00** to **2014-06-01 19:00:00** (corresponding to 6 days). Next, the best model is evaluated on the hold-out test data from \n", "**2014-06-01 20:00:00** to **2014-06-02 19:00:00** (corresponding the next 1 day). Finally, we show that the performance of model trained with HPO is significantly better than the one trained without HPO.\n", "\n", "\n", "This package depends on and incorporates or retrieves a third-party software package (such as an open source package) at install-time or build-time or run-time (\"External Dependency\"). The External Dependency is subject to a license term \n", "that you must accept in order to use this package. If you do not\n", "accept the applicable license term, you should not use this package. We\n", "recommend that you consult your company’s open source approval policy before\n", "proceeding.\n", "\n", "Provided below is the External Dependency and the applicable license\n", "identification as indicated by the documentation associated with the External\n", "Dependency as of Amazon's most recent review.\n", "\n", "Package: PyMeeus. License: [GNU Lesser General Public License v3 (LGPLv3) (LGPLv3)](https://github.com/architest/pymeeus/blob/master/LICENSE.txt).\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.tuner import (\n", " IntegerParameter,\n", " CategoricalParameter,\n", " ContinuousParameter,\n", " HyperparameterTuner,\n", ")\n", "\n", "# Static hyperparameters we do not tune\n", "hyperparameters = {\n", " \"context_length\": CONTEXT_LENGTH, # sliding window size for training\n", " \"prediction_length\": PREDICTION_LENGTH, # sliding window size for predicting\n", " \"skip_size\": 4, # skip size is used in skip-rnn layer\n", " \"ar_window\": 4, # auto-regressive window size\n", " \"scaling\": True, # whether to scale the data or not\n", " \"batch_size\": 32, # number of batch size\n", " \"rnn_cell_type\": \"gru\", # type of the RNN cell. Either lstm or gru (default: gru)\n", " \"rnn_num_layers\": 1, # number of RNN layers to be used\n", " \"rnn_num_cells\": 100, # number of RNN cells for each layer\n", " \"skip_rnn_cell_type\": \"gru\", # type of the RNN cell for the skip layer. Either lstm or gru\n", " \"skip_rnn_num_layers\": 1, # number of RNN layers to be used for skip part\n", " \"skip_rnn_num_cells\": 10, # number of RNN cells for each layer for skip part\n", " \"lead_time\": 0, # lead time\n", " \"kernel_size\": 6, # kernel size for first layer Conv2D\n", "}\n", "\n", "# Dynamic hyperparameters we want to tune and their searching ranges. For demonstartion purpose, we skip the architecture search by skipping tunning the hyperparameters such as 'skip_rnn_num_layers', 'rnn_num_layers', and etc.\n", "hyperparameter_ranges = {\n", " \"dropout_rate\": ContinuousParameter(0.3, 0.5),\n", " \"channels\": CategoricalParameter(\n", " [80, 84, 88, 92, 96]\n", " ), # Note. channels should be divisible by skip_size\n", " \"output_activation\": CategoricalParameter([\"sigmoid\", \"tanh\"]),\n", " \"learning_rate\": ContinuousParameter(0.001, 0.01),\n", " \"epochs\": IntegerParameter(40, 50),\n", "}" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Define the objective metric name, metric definiton (with regex pattern), and objective type for the tuning job." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "objective_metric_name = \"Root Relative Squared Error (RRSE)\"\n", "metric_definitions = [\n", " {\n", " \"Name\": \"Root Relative Squared Error (RRSE)\",\n", " \"Regex\": \"Root Relative Squared Error \\\\(RRSE\\\\): (\\\\S+)\",\n", " }\n", "] # Root Relative Squared Error (RSE):\n", "objective_type = \"Minimize\"" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "!cp lstnet/requirements_train.txt lstnet/requirements.txt\n", "\n", "estimator_tuning = MXNet(\n", " entry_point=\"train.py\",\n", " source_dir=\"lstnet\",\n", " role=role,\n", " instance_count=1,\n", " instance_type=training_instance_type, # we use CPU machine here as there are multiple training jobs being triggered. You can also change it back to GPU machine ml.g4dn.2xlarge\n", " framework_version=\"1.8.0\",\n", " py_version=\"py37\",\n", " hyperparameters=hyperparameters,\n", " output_path=train_output,\n", " code_location=code_location,\n", " sagemaker_session=session,\n", " enable_network_isolation=True, # Set enable_network_isolation=True to ensure a security running environment\n", " # container_log_level=logging.DEBUG, # display debug logs\n", " env={\"MMS_DEFAULT_RESPONSE_TIMEOUT\": \"1000\"},\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tuning_job_name = solution_prefix + unique_hash + \"-lstnet-tuning\"\n", "print(\n", " f\"You can go to SageMaker -> Training -> Hyperparameter tuning jobs -> a job name started with {tuning_job_name} to monitor HPO tuning status and details.\\n\"\n", " f\"Note. You are not able to successfully run the following cells until the tuning job completes. This step may take around 1 hour.\"\n", ")\n", "\n", "tuner = HyperparameterTuner(\n", " estimator_tuning, # using the estimator defined in Stage I\n", " objective_metric_name,\n", " hyperparameter_ranges,\n", " metric_definitions,\n", " max_jobs=20,\n", " max_parallel_jobs=3,\n", " objective_type=objective_type,\n", " base_tuning_job_name=tuning_job_name,\n", ")\n", "\n", "start_time = time.time()\n", "\n", "tuner.fit(input_data)\n", "\n", "hpo_training_job_time_duration = time.time() - start_time" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Find the tuning job name" ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "sm_client = boto3.Session().client(\"sagemaker\")\n", "\n", "tuning_job_name = tuner.latest_tuning_job.name\n", "tuning_job_name" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Checking the status of the tuning jobs: whether all of them are finished with 'Completed' status." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "tuning_job_result = sm_client.describe_hyper_parameter_tuning_job(\n", " HyperParameterTuningJobName=tuning_job_name\n", ")\n", "\n", "status = tuning_job_result[\"HyperParameterTuningJobStatus\"]\n", "if status != \"Completed\":\n", " print(\"Reminder: the tuning job has not been completed.\")\n", "\n", "job_count = tuning_job_result[\"TrainingJobStatusCounters\"][\"Completed\"]\n", "print(\"%d training jobs have completed\" % job_count)\n", "\n", "is_minimize = (\n", " tuning_job_result[\"HyperParameterTuningJobConfig\"][\"HyperParameterTuningJobObjective\"][\"Type\"]\n", " != \"Minimize\"\n", ")\n", "objective_name = tuning_job_result[\"HyperParameterTuningJobConfig\"][\n", " \"HyperParameterTuningJobObjective\"\n", "][\"MetricName\"]" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Once the tuning job finishes, we can bring in a table of metrics." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "tuner_analytics = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n", "\n", "full_df = tuner_analytics.dataframe()\n", "\n", "if len(full_df) > 0:\n", " df = full_df[full_df[\"FinalObjectiveValue\"] > -float(\"inf\")]\n", " if len(df) > 0:\n", " df = df.sort_values(\"FinalObjectiveValue\", ascending=True)\n", " print(\"Number of training jobs with valid objective: %d\" % len(df))\n", " print(\n", " {\n", " \"lowest\": min(df[\"FinalObjectiveValue\"]),\n", " \"highest\": max(df[\"FinalObjectiveValue\"]),\n", " }\n", " )\n", " pd.set_option(\"display.max_colwidth\", -1) # Don't truncate TrainingJobName\n", " else:\n", " print(\"No training jobs have reported valid results yet.\")\n", "\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Get the output path for the best model from the HPO tuning." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df = df[df[\"TrainingJobStatus\"] == \"Completed\"] # filter out the failed jobs\n", "output_path_best_tuning_job = os.path.join(train_output, df[\"TrainingJobName\"].iloc[0], \"output\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "print(f\"The output path of the best model from the hpo tuning is: {output_path_best_tuning_job}\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "%%time\n", "!cp lstnet/requirements_inference.txt lstnet/requirements.txt\n", "\n", "endpoint_name = solution_prefix + unique_hash + \"-lstnet-tuning-endpoint\"\n", "print(\n", " f\"You can go to SageMaker -> Inference -> Endpoints --> an endpoint with name {endpoint_name} to monitor the deployment status.\"\n", ")\n", "\n", "tuning_best_model = MXNetModel(\n", " model_data=os.path.join(output_path_best_tuning_job, \"model.tar.gz\"),\n", " role=role,\n", " entry_point=\"inference.py\",\n", " source_dir=\"lstnet\",\n", " py_version=\"py37\",\n", " name=solution_prefix + \"-hpo-tuning-model\",\n", " framework_version=\"1.8.0\",\n", " code_location=code_location,\n", " enable_network_isolation=True, # Set enable_network_isolation=True to ensure a security running environment\n", " env={\"MMS_DEFAULT_RESPONSE_TIMEOUT\": \"1000\"},\n", ")\n", "\n", "tuning_predictor = tuning_best_model.deploy(\n", " instance_type=inference_instance_type,\n", " endpoint_name=endpoint_name,\n", " initial_instance_count=1,\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After deploying the endpoint, we query the endpoint using the same test data as defined in training LSTNet without HPO, compute the evaluaton metrics, and compare with the results without HPO tuning." ] }, { "cell_type": "code", "execution_count": null, "metadata": { "collapsed": false, "jupyter": { "outputs_hidden": false }, "pycharm": { "name": "#%%\n" } }, "outputs": [], "source": [ "start_time = time.time()\n", "predictions_tuning = tuning_predictor.predict(test_data)\n", "predicted_lstnet = predictions_tuning[\"forecasts\"][\"samples\"]\n", "inference_time_duration_tuning = time.time() - start_time" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "dict_agg_metrics_with_hpo = json.loads(predictions_tuning[\"agg_metrics\"])\n", "dict_agg_metrics_with_hpo[\"RRSE\"] = dict_agg_metrics_with_hpo[\"RMSE\"] / np.std(\n", " test_data[\"target\"]\n", ") # compute Root Relative Squared Error (RRSE)\n", "dict_agg_metrics_with_hpo[\"Training Time (min)\"] = hpo_training_job_time_duration / 60\n", "dict_agg_metrics_with_hpo[\"Inference Time (s)\"] = inference_time_duration_tuning\n", "df_agg_metrics_with_hpo = pd.DataFrame.from_dict(dict_agg_metrics_with_hpo, orient=\"index\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# Combine with the evaluation metrics without hpo tuning\n", "df_agg_metrics_with_hpo_eval_metrics = df_agg_metrics_with_hpo.reindex(METRICS_TO_SHOW)\n", "df_agg_metrics_with_hpo_eval_metrics.columns = [\"LSTNet with HPO\"]\n", "df_agg_metrics_without_hpo_eval_metrics.join(df_agg_metrics_with_hpo_eval_metrics)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see the metrics values with HPO tuning is smaller than those without HPO tuning on the same test data. This indicates that HPO tuning further improves the model performance. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stage III: Train an Optimal SageMaker DeepAR Model with HPO.\n", "\n", "The [Amazon SageMaker DeepAR](https://docs.aws.amazon.com/sagemaker/latest/dg/deepar.html) forecasting algorithm is a supervised learning algorithm for forecasting scalar (one-dimensional) time series using recurrent neural networks (RNN). Classical forecasting methods, such as autoregressive integrated moving average (ARIMA) or exponential smoothing (ETS), fit a single model to each individual time series. They then use that model to extrapolate the time series into the future.\n", "\n", "In many applications, however, you have many similar time series across a set of cross-sectional units. For example, you might have time series groupings for demand for different products, server loads, and requests for webpages. For this type of application, you can benefit from training a single model jointly over all of the time series. DeepAR takes this approach. When your dataset contains hundreds of related time series, DeepAR outperforms the standard ARIMA and ETS methods. You can also use the trained model to generate forecasts for new time series that are similar to the ones it has been trained on.\n", "\n", "For information on the mathematics behind DeepAR, see [DeepAR: Probabilistic Forecasting with Autoregressive Recurrent Networks](https://arxiv.org/abs/1704.04110).\n", "\n", "Similar to the setting in the previous stages, we do following:\n", "\n", "* We firstly train a DeepAR model without HPO. \n", "* Next, we train an optimal DeepAR model with HPO. \n", "* We demonstrate that the DeepAR trained with HPO significantly outperform the one trained without HPO." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Prepare the traing, validation, and test data" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from typing import Tuple, Optional, List\n", "from datetime import timedelta\n", "from gluonts.dataset.rolling_dataset import StepStrategy, generate_rolling_dataset\n", "from gluonts.dataset.multivariate_grouper import MultivariateGrouper\n", "from gluonts.dataset.common import TrainDatasets, load_datasets" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Read the data from local directory `raw_data`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "path = Path(\"raw_data\")\n", "metadata_path = path if path == Path(\"raw_data\") else path / \"metadata\"\n", "ds = load_datasets(metadata_path, path / \"train\", path / \"test\")\n", "target_dim = int(ds.metadata.feat_static_cat[0].cardinality)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "freq = ds.metadata.freq\n", "ts_first_dim = next(iter(ds.train))\n", "start_dataset = ts_first_dim[\"start\"]\n", "ts_train_len = ts_first_dim[\"target\"].shape[0]\n", "end_dataset = pd.date_range(start_dataset, periods=ts_train_len, freq=freq)[-1]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Reformat the training data as input for DeepAR." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "training_data = [\n", " {\n", " \"start\": str(start_dataset),\n", " \"target\": ts[\n", " \"target\"\n", " ].tolist(), # We use -1, because pandas indexing includes the upper bound\n", " }\n", " for ts in ds.train\n", "]\n", "print(f\"Train data contains {len(training_data)} time series.\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Since the first 6 days of test data are used as the valiation set (same setting as in the previous evaluation), we generate validation data that extends to 1, 2, 3, 4, 5, 6 days beyond the training range. This way we perform rolling evaluation of our model." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "num_valid_windows = 6\n", "\n", "validation_data = [\n", " {\n", " \"start\": str(ts[\"start\"]),\n", " \"target\": ts[\"target\"][: (ts_train_len + k * PREDICTION_LENGTH)].tolist(),\n", " }\n", " for k in range(1, num_valid_windows + 1)\n", " for ts in list(ds.test)[-target_dim:]\n", "]\n", "print(f\"The validation data contains {len(validation_data)} time series.\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def write_dicts_to_file(path, data):\n", " with open(path, \"wb\") as fp:\n", " for d in data:\n", " fp.write(json.dumps(d).encode(\"utf-8\"))\n", " fp.write(\"\\n\".encode(\"utf-8\"))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Write the train and validation file into local directory `input_data_deepar`" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "!mkdir -p input_data_deepar\n", "\n", "write_dicts_to_file(\"input_data_deepar/train.json\", training_data)\n", "write_dicts_to_file(\"input_data_deepar/validation.json\", validation_data)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Then we copy them to the S3 bucket where the DeepAR reads the input data from." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s3 = boto3.resource(\"s3\")\n", "\n", "\n", "def copy_to_s3(local_file, s3_path, override=True):\n", " assert s3_path.startswith(\"s3://\")\n", " split = s3_path.split(\"/\")\n", " bucket = split[2]\n", " path = \"/\".join(split[3:])\n", " buk = s3.Bucket(bucket)\n", "\n", " if len(list(buk.objects.filter(Prefix=path))) > 0:\n", " if not override:\n", " print(\n", " \"File s3://{}/{} already exists.\\nSet override to upload anyway.\\n\".format(\n", " s3_bucket, s3_path\n", " )\n", " )\n", " return\n", " else:\n", " print(\"Overwriting existing file\")\n", " with open(local_file, \"rb\") as data:\n", " print(\"Uploading file to {}\".format(s3_path))\n", " buk.put_object(Key=path, Body=data)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "s3_input_path_deepar = f\"s3://{bucket}/{prefix}/input_data_deepar\"\n", "s3_output_path_deepar = f\"s3://{bucket}/{prefix}/output_deepar\"\n", "\n", "copy_to_s3(\"input_data_deepar/train.json\", s3_input_path_deepar + \"/train/train.json\")\n", "copy_to_s3(\n", " \"input_data_deepar/validation.json\",\n", " s3_input_path_deepar + \"/validation/validation.json\",\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Train a SageMaker DeepAR Model without HPO\n" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "import boto3\n", "\n", "boto3_session = boto3.session.Session()\n", "region = boto3_session.region_name\n", "image_name = sagemaker.amazon.amazon_estimator.get_image_uri(region, \"forecasting-deepar\", \"latest\")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "training_job_name_deepar = solution_prefix + unique_hash + \"-deepar-training\"\n", "\n", "deepar_estimator = sagemaker.estimator.Estimator(\n", " image_uri=image_name,\n", " sagemaker_session=session,\n", " role=role,\n", " train_instance_count=1, # change it to 2 for distributed training to accelerate training\n", " train_instance_type=training_instance_type, # we use a GPU machine here to shorten HPO runtime as there are multiple training jobs being triggered. You can also change it to a CPU machine 'ml.m5.4xlarge'.\n", " base_job_name=training_job_name_deepar,\n", " output_path=s3_output_path_deepar,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hyperparameters = {\n", " \"time_freq\": freq,\n", " \"epochs\": \"400\",\n", " \"early_stopping_patience\": \"40\",\n", " \"mini_batch_size\": \"64\",\n", " \"learning_rate\": \"5E-4\",\n", " \"context_length\": str(CONTEXT_LENGTH),\n", " \"prediction_length\": str(PREDICTION_LENGTH),\n", "}" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deepar_estimator.set_hyperparameters(**hyperparameters)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "data_channels = {\n", " \"train\": \"{}/train/\".format(s3_input_path_deepar),\n", " \"test\": \"{}/validation/\".format(s3_input_path_deepar),\n", "}\n", "\n", "print(\n", " f\"You can go to SageMaker -> Training -> Training jobs -> a job name started with {training_job_name_deepar} to monitor training status and details.\"\n", ")\n", "\n", "start_time = time.time()\n", "deepar_estimator.fit(inputs=data_channels, wait=True, logs=False)\n", "deepar_training_job_time_duration = time.time() - start_time" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deploy the Endpoint" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "from sagemaker.serializers import IdentitySerializer\n", "from sagemaker.deserializers import JSONDeserializer" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Following utility class allows for using `pandas.Series` objects to query the endpoint." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "def series_to_obj(ts, cat=None):\n", " obj = {\"start\": str(ts.index[0]), \"target\": list(ts)}\n", " if cat:\n", " obj[\"cat\"] = cat\n", " return obj\n", "\n", "\n", "class DeepARPredictor(sagemaker.predictor.Predictor):\n", " def __init__(self, *args, **kwargs):\n", " super().__init__(\n", " *args,\n", " serializer=IdentitySerializer(content_type=\"application/json\"),\n", " **kwargs,\n", " )\n", "\n", " def set_prediction_parameters(self, freq, prediction_length):\n", " \"\"\"Set the time frequency and prediction length parameters. This method **must** be\n", " called before being able to use `predict`.\n", "\n", " Parameters:\n", " freq -- string indicating the time frequency\n", " prediction_length -- integer, number of predicted time points\n", "\n", " Return value: none.\n", " \"\"\"\n", " self.freq = freq\n", " self.prediction_length = prediction_length\n", "\n", " def predict(\n", " self,\n", " ts,\n", " cat=None,\n", " encoding=\"utf-8\",\n", " num_samples=100,\n", " quantiles=[\"0.1\", \"0.5\", \"0.9\"],\n", " ):\n", " \"\"\"Requests the prediction of for the time series listed in `ts`, each with the\n", " (optional) corresponding category listed in `cat`.\n", "\n", " Parameters:\n", " ts -- list of `pandas.Series` objects, the time series to predict\n", " cat -- list of integers (default: None)\n", " encoding -- string, encoding to use for the request (default: 'utf-8')\n", " num_samples -- integer, number of samples to compute at prediction time (default: 100)\n", " quantiles -- list of strings specifying the quantiles to compute (default: ['0.1', '0.5', '0.9'])\n", "\n", " Return value: list of `pandas.DataFrame` objects, each containing the predictions\n", " \"\"\"\n", " prediction_times = [x.index[-1] + x.index.freq for x in ts]\n", " req = self.__encode_request(ts, cat, encoding, num_samples, quantiles)\n", " res = super(DeepARPredictor, self).predict(req)\n", " return self.__decode_response(res, prediction_times, encoding)\n", "\n", " def __encode_request(self, ts, cat, encoding, num_samples, quantiles):\n", " instances = [series_to_obj(ts[k], cat[k] if cat else None) for k in range(len(ts))]\n", " configuration = {\n", " \"num_samples\": num_samples,\n", " \"output_types\": [\"quantiles\", \"mean\"],\n", " \"quantiles\": quantiles,\n", " }\n", " http_request_data = {\"instances\": instances, \"configuration\": configuration}\n", " return json.dumps(http_request_data).encode(encoding)\n", "\n", " def __decode_response(self, response, prediction_times, encoding):\n", " response_data = json.loads(response.decode(encoding))\n", " list_of_df = []\n", " for k in range(len(prediction_times)):\n", " prediction_index = pd.date_range(\n", " start=prediction_times[k],\n", " freq=self.freq,\n", " periods=self.prediction_length,\n", " )\n", " res_df = pd.DataFrame(\n", " data=response_data[\"predictions\"][k][\"quantiles\"],\n", " index=prediction_index,\n", " )\n", " sample_mean_predict_df = pd.DataFrame(\n", " {\n", " \"mean\": response_data[\"predictions\"][k][\"mean\"],\n", " },\n", " index=prediction_index,\n", " )\n", " res_df = pd.concat([res_df, sample_mean_predict_df], axis=1)\n", " list_of_df.append(res_df)\n", " return list_of_df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Different from GluonTS algorithms, the test data should not contain the values you want to predict." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "test_data = [\n", " pd.Series(\n", " ts[\"target\"][:-PREDICTION_LENGTH].tolist(),\n", " index=pd.date_range(\n", " start=ts[\"start\"],\n", " periods=ts[\"target\"][:-PREDICTION_LENGTH].shape[0],\n", " freq=freq,\n", " ),\n", " )\n", " for ts in list(ds.test)[-target_dim:]\n", "]\n", "print(len(test_data))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "endpoint_name_deepar = solution_prefix + unique_hash + \"-deepar-endpoint\"\n", "print(\n", " f\"You can go to SageMaker -> Inference -> Endpoints --> an endpoint with name {endpoint_name_deepar} to monitor the deployment status.\"\n", ")\n", "\n", "deepar_endpoint = deepar_estimator.deploy(\n", " initial_instance_count=1,\n", " endpoint_name=endpoint_name_deepar,\n", " instance_type=inference_instance_type,\n", " serializer=IdentitySerializer(content_type=\"application/json\"),\n", " deserializer=JSONDeserializer(),\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deepar_predictor = DeepARPredictor(\n", " endpoint_name=deepar_endpoint.endpoint_name, sagemaker_session=session\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "After the endpoint is deployed, we query the endpoint to get model predictions for the test data." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "start_time = time.time()\n", "\n", "deepar_predictor.set_prediction_parameters(freq, PREDICTION_LENGTH)\n", "predictions = []\n", "for i in range(len(test_data)):\n", " predictions.extend(deepar_predictor.predict(test_data[i : (i + 1)], num_samples=100))\n", "\n", "deepar_inference_time_duration = time.time() - start_time" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "forecasts_all = []\n", "forecasts_all.extend(\n", " [\n", " SampleForecast(\n", " samples=np.expand_dims(np.array(x[\"mean\"]), axis=0),\n", " start_date=x[\"mean\"].index[0],\n", " freq=freq,\n", " )\n", " for x in predictions\n", " ]\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Compute the evaluation metrics." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "evaluator = Evaluator()\n", "tss_all = [\n", " pd.Series(\n", " ts[\"target\"].tolist(),\n", " index=pd.date_range(start=ts[\"start\"], periods=ts[\"target\"].shape[0], freq=freq),\n", " )\n", " for ts in list(ds.test)[-target_dim:]\n", "]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "agg_metrics, item_metrics = evaluator(iter(tss_all), iter(forecasts_all), num_series=len(test_data))" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deepar_dict_agg_metrics_without_hpo = agg_metrics\n", "deepar_dict_agg_metrics_without_hpo[\"RRSE\"] = deepar_dict_agg_metrics_without_hpo[\"RMSE\"] / np.std(\n", " tss_all\n", ") # compute Root Relative Squared Error (RRSE)\n", "deepar_dict_agg_metrics_without_hpo[\"Training Time (min)\"] = deepar_training_job_time_duration / 60\n", "deepar_dict_agg_metrics_without_hpo[\"Inference Time (s)\"] = deepar_inference_time_duration\n", "deepar_dict_agg_metrics_without_hpo = pd.DataFrame.from_dict(\n", " deepar_dict_agg_metrics_without_hpo, orient=\"index\", columns=[\"DeepAR without HPO\"]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deepar_dict_agg_metrics_without_hpo = deepar_dict_agg_metrics_without_hpo.reindex(METRICS_TO_SHOW)\n", "deepar_dict_agg_metrics_without_hpo" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Train an Optimal SageMaker DeepAR with HPO" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deepar = sagemaker.estimator.Estimator(\n", " image_uri=image_name,\n", " sagemaker_session=session,\n", " role=role,\n", " train_instance_count=1,\n", " train_instance_type=training_instance_type, # we use a GPU machine here to shorten HPO runtime as there are multiple training jobs being triggered. You can also change it to a CPU machine 'ml.m5.2xlarge'.\n", " output_path=s3_output_path_deepar,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hyperparameters = {\n", " \"time_freq\": freq,\n", " \"early_stopping_patience\": \"40\",\n", " \"context_length\": str(CONTEXT_LENGTH),\n", " \"prediction_length\": str(PREDICTION_LENGTH),\n", "}\n", "\n", "deepar.set_hyperparameters(**hyperparameters)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "hyperparameter_ranges = {\n", " \"mini_batch_size\": IntegerParameter(100, 400),\n", " \"epochs\": IntegerParameter(200, 400),\n", " \"num_cells\": IntegerParameter(50, 100),\n", " \"likelihood\": CategoricalParameter([\"negative-binomial\", \"student-T\"]),\n", " \"learning_rate\": ContinuousParameter(0.001, 0.1),\n", "}" ] }, { "cell_type": "markdown", "metadata": { "pycharm": { "name": "#%% md\n" } }, "source": [ "Note. To reduce HPO runtime, please adjust parameter `max_parallel_jobs` in function `HyperparameterTuner` below, depending on number of available instances in your account. Current `max_parallel_jobs` is set as 3." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "objective_metric_name = \"test:RMSE\" # RMSE * (standard deviation of training data) = RRSE, since the standard deviation of training data is fixed. Optimizing RMSE is the same as optimizing RRSE.\n", "\n", "tuning_job_name_deepar = solution_prefix + unique_hash + \"-deepar-tuning\"\n", "print(\n", " f\"You can go to SageMaker -> Training -> Hyperparameter tuning jobs -> a job name started with {tuning_job_name_deepar} to monitor HPO tuning status and details.\\n\"\n", " f\"Note. You are not able to successfully run the following cells until the tuning job completes. This step may take up to 2-3 hours depending on parameter max_parallel_jobs.\"\n", ")\n", "\n", "start_time = time.time()\n", "deepar_tuner = HyperparameterTuner(\n", " deepar,\n", " objective_metric_name,\n", " hyperparameter_ranges,\n", " max_jobs=20,\n", " strategy=\"Bayesian\",\n", " objective_type=\"Minimize\",\n", " max_parallel_jobs=3,\n", " early_stopping_type=\"Auto\",\n", " base_tuning_job_name=tuning_job_name_deepar,\n", ")\n", "\n", "deepar_tuner.fit(data_channels, include_cls_metadata=False)\n", "\n", "deepar_training_job_time_duration_hpo = time.time() - start_time" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Examine the HPO tuning job results" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Find the tuning job name" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "sm_client = boto3.Session().client(\"sagemaker\")\n", "\n", "tuning_job_name = deepar_tuner.latest_tuning_job.name\n", "tuning_job_name" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Checking the status of the tuning jobs: whether all of them are finished with 'Completed' status." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tuning_job_result = sm_client.describe_hyper_parameter_tuning_job(\n", " HyperParameterTuningJobName=tuning_job_name\n", ")\n", "\n", "status = tuning_job_result[\"HyperParameterTuningJobStatus\"]\n", "if status != \"Completed\":\n", " print(\"Reminder: the tuning job has not been completed.\")\n", "\n", "job_count = tuning_job_result[\"TrainingJobStatusCounters\"][\"Completed\"]\n", "print(\"%d training jobs have completed\" % job_count)\n", "\n", "is_minimize = (\n", " tuning_job_result[\"HyperParameterTuningJobConfig\"][\"HyperParameterTuningJobObjective\"][\"Type\"]\n", " != \"Minimize\"\n", ")\n", "objective_name = tuning_job_result[\"HyperParameterTuningJobConfig\"][\n", " \"HyperParameterTuningJobObjective\"\n", "][\"MetricName\"]" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Once the tuning job finishes, we can bring in a table of metrics." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "tuner_analytics = sagemaker.HyperparameterTuningJobAnalytics(tuning_job_name)\n", "\n", "full_df = tuner_analytics.dataframe()\n", "\n", "if len(full_df) > 0:\n", " df = full_df[full_df[\"FinalObjectiveValue\"] > -float(\"inf\")]\n", " if len(df) > 0:\n", " df = df.sort_values(\"FinalObjectiveValue\", ascending=True)\n", " print(\"Number of training jobs with valid objective: %d\" % len(df))\n", " print(\n", " {\n", " \"lowest\": min(df[\"FinalObjectiveValue\"]),\n", " \"highest\": max(df[\"FinalObjectiveValue\"]),\n", " }\n", " )\n", " pd.set_option(\"display.max_colwidth\", -1) # Don't truncate TrainingJobName\n", " else:\n", " print(\"No training jobs have reported valid results yet.\")\n", "\n", "df" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Deploy an Endpoint for the Best Tuning Job" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "endpoint_name_deepar_hpo = solution_prefix + unique_hash + \"-deepar-tuning-endpoint\"\n", "print(\n", " f\"You can go to SageMaker -> Inference -> Endpoints --> an endpoint with name {endpoint_name_deepar_hpo} to monitor the deployment status.\"\n", ")\n", "\n", "deepar_endpoint_hpo = deepar_tuner.deploy(\n", " initial_instance_count=1,\n", " endpoint_name=endpoint_name_deepar_hpo,\n", " instance_type=inference_instance_type,\n", " serializer=IdentitySerializer(content_type=\"application/json\"),\n", " deserializer=JSONDeserializer(),\n", " wait=True,\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deepar_predictor_hpo = DeepARPredictor(\n", " endpoint_name=deepar_endpoint_hpo.endpoint_name, sagemaker_session=session\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "start_time = time.time()\n", "\n", "deepar_predictor_hpo.set_prediction_parameters(freq, PREDICTION_LENGTH)\n", "predictions_hpo = []\n", "for i in range(len(test_data)):\n", " predictions_hpo.extend(deepar_predictor_hpo.predict(test_data[i : (i + 1)], num_samples=100))\n", "\n", "deepar_inference_time_duration_hpo = time.time() - start_time" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "forecasts_all_hpo = []\n", "forecasts_all_hpo.extend(\n", " [\n", " SampleForecast(\n", " samples=np.expand_dims(np.array(x[\"mean\"]), axis=0),\n", " start_date=x[\"mean\"].index[0],\n", " freq=freq,\n", " )\n", " for x in predictions_hpo\n", " ]\n", ")\n", "predicted_deepar = forecasts_all_hpo" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "agg_metrics_hpo, item_metrics_hpo = evaluator(\n", " iter(tss_all), iter(forecasts_all_hpo), num_series=len(test_data)\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deepar_dict_agg_metrics_with_hpo = agg_metrics_hpo\n", "deepar_dict_agg_metrics_with_hpo[\"RRSE\"] = deepar_dict_agg_metrics_with_hpo[\"RMSE\"] / np.std(\n", " tss_all\n", ") # compute Root Relative Squared Error (RRSE)\n", "deepar_dict_agg_metrics_with_hpo[\"Training Time (min)\"] = deepar_training_job_time_duration_hpo / 60\n", "deepar_dict_agg_metrics_with_hpo[\"Inference Time (s)\"] = deepar_inference_time_duration_hpo\n", "deepar_dict_agg_metrics_with_hpo = pd.DataFrame.from_dict(\n", " deepar_dict_agg_metrics_with_hpo, orient=\"index\", columns=[\"DeepAR with HPO\"]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "deepar_results_all = pd.concat(\n", " [\n", " deepar_dict_agg_metrics_without_hpo,\n", " deepar_dict_agg_metrics_with_hpo.reindex(METRICS_TO_SHOW),\n", " ],\n", " axis=1,\n", ")\n", "deepar_results_all" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can see the metrics values with HPO tuning is smaller than those without HPO tuning on the same test data. This indicates that HPO tuning further improves the model performance. " ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Stage IV: Evaluate Model Performance of All Three Algorithms on the Same Hold-out Test Data." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Compare the Model Performance from the Three Models Trained from HPO\n", "\n", "Except for the training and inference time, for RRSE (Root Relative Squared Error), MAPE (Mean Absolute Percentage Error), and sMAPE (symmetric Mean Absolute Percentage Error), smaller values indicate better predictive performance." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "df_agg_metrics_with_hpo_eval_metrics.join(\n", " deepar_dict_agg_metrics_with_hpo\n", ")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "### Visualize Model Predictions" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "timestamps_test = pd.date_range(test_df_viz.iloc[0][\"time\"], periods=CONTEXT_LENGTH, freq=meta.freq)\n", "ts_start, ts_end = timestamps_test[0], timestamps_test[-1]\n", "\n", "\n", "def get_preds_df_filter(forecasts: np.array):\n", " preds = ListDataset(\n", " [\n", " {\n", " FieldName.TARGET: forecasts,\n", " FieldName.START: ts_start.strftime(\"%Y-%m-%d %H:%M:%S\"),\n", " }\n", " ],\n", " freq=meta.freq,\n", " one_dim_target=False,\n", " )\n", " preds_df = multivar_df(next(iter(preds)))\n", " preds_df_filter = preds_df.loc[:, [\"time\"] + selected_cols]\n", " preds_df_filter = pd.melt(preds_df_filter, id_vars=[\"time\"], value_vars=selected_cols)\n", " preds_df_filter.rename(columns={\"variable\": \"covariate\"}, inplace=True)\n", " return preds_df_filter\n", "\n", "\n", "lstnet_preds_df_filter = get_preds_df_filter(np.transpose(np.array(predicted_lstnet[0])))\n", "deepar_preds_df_filter = get_preds_df_filter(np.array([x.samples[0] for x in predicted_deepar]))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Set the time series indexes that we want to plot" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "selected_ts = [\n", " \"ts_98\",\n", " \"ts_100\",\n", " \"ts_116\",\n", " \"ts_137\",\n", " \"ts_147\",\n", " \"ts_156\",\n", " \"ts_184\",\n", " \"ts_216\",\n", " \"ts_245\",\n", " \"ts_267\",\n", "]" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "selection = alt.selection_multi(fields=[\"covariate\"], bind=\"legend\", nearest=True)\n", "\n", "train_plot = (\n", " alt.Chart(\n", " train_df_viz.loc[train_df_viz[\"covariate\"].isin(selected_ts)],\n", " title=\"Input data\",\n", " )\n", " .mark_line()\n", " .encode(\n", " alt.X(\"time:T\", axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\")),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 3000]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "gt_plot = (\n", " alt.Chart(\n", " test_df_viz.loc[test_df_viz[\"covariate\"].isin(selected_ts)],\n", " title=\"Expected output data (ground truth)\",\n", " )\n", " .mark_line()\n", " .encode(\n", " alt.X(\n", " \"time:T\",\n", " axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\"),\n", " scale=alt.Scale(domain=[ts_start, ts_end]),\n", " ),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 3500]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "lstnet_preds_plot = (\n", " alt.Chart(\n", " lstnet_preds_df_filter.loc[lstnet_preds_df_filter[\"covariate\"].isin(selected_ts)],\n", " title=\"LSTNet predictions\",\n", " )\n", " .mark_line()\n", " .encode(\n", " alt.X(\n", " \"time:T\",\n", " axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\"),\n", " scale=alt.Scale(domain=[ts_start, ts_end]),\n", " ),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 3500]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "\n", "deepar_preds_plot = (\n", " alt.Chart(\n", " deepar_preds_df_filter.loc[deepar_preds_df_filter[\"covariate\"].isin(selected_ts)],\n", " title=\"DeepAR predictions\",\n", " )\n", " .mark_line()\n", " .encode(\n", " alt.X(\n", " \"time:T\",\n", " axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\"),\n", " scale=alt.Scale(domain=[ts_start, ts_end]),\n", " ),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 3500]),\n", " ),\n", " alt.Color(\"covariate:N\"),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", ")\n", "\n", "(train_plot | gt_plot) & (lstnet_preds_plot) & deepar_preds_plot" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Another way to visualize the results" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "train_and_test_df_viz = pd.concat([train_df_viz, test_df_viz])\n", "train_and_test_df_viz[\"Type\"] = \"Ground truth\"\n", "lstnet_preds_df_filter[\"Type\"] = \"LSTNet\"\n", "deepar_preds_df_filter[\"Type\"] = \"DeepAR\"\n", "merge_preds_df_filter = pd.concat(\n", " [\n", " train_and_test_df_viz,\n", " lstnet_preds_df_filter,\n", " deepar_preds_df_filter,\n", " ]\n", ")" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "selected_ts = [\"ts_100\"]\n", "selection = alt.selection_multi(fields=[\"Type\"], bind=\"legend\", nearest=True)\n", "merge_preds_plot = (\n", " alt.Chart(\n", " merge_preds_df_filter.loc[merge_preds_df_filter[\"covariate\"].isin(selected_ts)],\n", " title=f'All predictions for \"{selected_ts[0]}\"',\n", " )\n", " .mark_line()\n", " .encode(\n", " alt.X(\"time:T\", axis=alt.Axis(title=\"Time\", format=\"%e %b, %y\")),\n", " alt.Y(\n", " \"value:Q\",\n", " axis=alt.Axis(title=\"Electricity consumption (kW)\"),\n", " scale=alt.Scale(domain=[0, 3500]),\n", " ),\n", " alt.Color(\n", " \"Type\",\n", " scale=alt.Scale(\n", " domain=[\"Ground truth\", \"LSTNet\", \"DeepAR\"],\n", " range=[\"black\", \"brown\", \"blue\", \"green\"],\n", " ),\n", " ),\n", " strokeDash=alt.condition(\n", " alt.datum.Type != \"Ground truth\",\n", " alt.value([5, 3]),\n", " alt.value([0]),\n", " ),\n", " strokeWidth=alt.condition(alt.datum.Type != \"Ground truth\", alt.value(1.5), alt.value(2)),\n", " opacity=alt.condition(selection, alt.value(1), alt.value(0.1)),\n", " )\n", " .add_selection(selection)\n", " .properties(width=800, height=400)\n", ")\n", "rules = (\n", " alt.Chart(pd.DataFrame({\"Date\": [ts_start.strftime(\"%Y-%m-%d %H:%M:%S\")], \"color\": [\"red\"]}))\n", " .mark_rule()\n", " .encode(\n", " x=\"Date:T\",\n", " color=alt.Color(\"color:N\", scale=None, legend=alt.Legend(title=\"Type\")),\n", " strokeWidth=alt.value(2),\n", " )\n", ")\n", "\n", "merge_preds_plot + rules" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The training and test data (ground truth) are shown as the black solid line (seperated by red vertical line) in the plot. The predictions from different forecasting algorithms are shown as dash lines. The closer the dash line comes to the black solid line, the more accurate the predictions are." ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Clean Up the Resources\n", "\n", "After you are done using this notebook, delete the model and the endpoint to avoid any incurring charges.\n", "\n", "**Caution**: You need to manually delete resources that you may have created while running the notebook, such as Amazon S3 buckets for model artifacts, training datasets, processing artifacts, and Amazon CloudWatch log groups." ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "predictor.delete_model() # LSTNet without HPO\n", "predictor.delete_endpoint() # LSTNet without HPO\n", "\n", "tuning_predictor.delete_model() # LSTNet with HPO\n", "tuning_predictor.delete_endpoint() # LSTNet with HPO\n", "\n", "deepar_predictor.delete_model() # DeepAR without HPO\n", "deepar_predictor.delete_endpoint() # DeepAR without HPO\n", "\n", "deepar_predictor_hpo.delete_model() # DeepAR with HPO\n", "deepar_predictor_hpo.delete_endpoint() # DeepAR with HPO" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Notebook CI Test Results\n", "\n", "This notebook was tested in multiple regions. The test results are as follows, except for us-west-2 which is shown at the top of the notebook.\n", "\n", "![This us-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This us-east-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-east-2/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This us-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/us-west-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This ca-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ca-central-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This sa-east-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/sa-east-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This eu-west-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This eu-west-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-2/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This eu-west-3 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-west-3/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This eu-central-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-central-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This eu-north-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/eu-north-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This ap-southeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This ap-southeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-southeast-2/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This ap-northeast-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This ap-northeast-2 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-northeast-2/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n", "\n", "![This ap-south-1 badge failed to load. Check your device's internet connectivity, otherwise the service is currently unavailable](https://prod.us-west-2.tcx-beacon.docs.aws.dev/sagemaker-nb/ap-south-1/introduction_to_applying_machine_learning|deep_demand_forecasting|deep_demand_forecast.ipynb)\n" ] } ], "metadata": { "availableInstances": [ { "_defaultOrder": 0, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.t3.medium", "vcpuNum": 2 }, { "_defaultOrder": 1, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.t3.large", "vcpuNum": 2 }, { "_defaultOrder": 2, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.t3.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 3, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.t3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 4, "_isFastLaunch": true, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5.large", "vcpuNum": 2 }, { "_defaultOrder": 5, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 6, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 7, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 8, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 9, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 10, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 11, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 12, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.m5d.large", "vcpuNum": 2 }, { "_defaultOrder": 13, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.m5d.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 14, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.m5d.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 15, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.m5d.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 16, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.m5d.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 17, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.m5d.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 18, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.m5d.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 19, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.m5d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 20, "_isFastLaunch": false, "category": "General purpose", "gpuNum": 0, "hideHardwareSpecs": true, "memoryGiB": 0, "name": "ml.geospatial.interactive", "supportedImageNames": [ "sagemaker-geospatial-v1-0" ], "vcpuNum": 0 }, { "_defaultOrder": 21, "_isFastLaunch": true, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 4, "name": "ml.c5.large", "vcpuNum": 2 }, { "_defaultOrder": 22, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 8, "name": "ml.c5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 23, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.c5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 24, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.c5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 25, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 72, "name": "ml.c5.9xlarge", "vcpuNum": 36 }, { "_defaultOrder": 26, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 96, "name": "ml.c5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 27, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 144, "name": "ml.c5.18xlarge", "vcpuNum": 72 }, { "_defaultOrder": 28, "_isFastLaunch": false, "category": "Compute optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.c5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 29, "_isFastLaunch": true, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g4dn.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 30, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g4dn.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 31, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g4dn.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 32, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g4dn.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 33, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g4dn.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 34, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g4dn.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 35, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 61, "name": "ml.p3.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 36, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 244, "name": "ml.p3.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 37, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 488, "name": "ml.p3.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 38, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.p3dn.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 39, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.r5.large", "vcpuNum": 2 }, { "_defaultOrder": 40, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.r5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 41, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.r5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 42, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.r5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 43, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.r5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 44, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.r5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 45, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 512, "name": "ml.r5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 46, "_isFastLaunch": false, "category": "Memory Optimized", "gpuNum": 0, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.r5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 47, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 16, "name": "ml.g5.xlarge", "vcpuNum": 4 }, { "_defaultOrder": 48, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 32, "name": "ml.g5.2xlarge", "vcpuNum": 8 }, { "_defaultOrder": 49, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 64, "name": "ml.g5.4xlarge", "vcpuNum": 16 }, { "_defaultOrder": 50, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 128, "name": "ml.g5.8xlarge", "vcpuNum": 32 }, { "_defaultOrder": 51, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 1, "hideHardwareSpecs": false, "memoryGiB": 256, "name": "ml.g5.16xlarge", "vcpuNum": 64 }, { "_defaultOrder": 52, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 192, "name": "ml.g5.12xlarge", "vcpuNum": 48 }, { "_defaultOrder": 53, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 4, "hideHardwareSpecs": false, "memoryGiB": 384, "name": "ml.g5.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 54, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 768, "name": "ml.g5.48xlarge", "vcpuNum": 192 }, { "_defaultOrder": 55, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4d.24xlarge", "vcpuNum": 96 }, { "_defaultOrder": 56, "_isFastLaunch": false, "category": "Accelerated computing", "gpuNum": 8, "hideHardwareSpecs": false, "memoryGiB": 1152, "name": "ml.p4de.24xlarge", "vcpuNum": 96 } ], "instance_type": "ml.t3.medium", "kernelspec": { "display_name": "Python 3 (Data Science 3.0)", "language": "python", "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/sagemaker-data-science-310-v1" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" } }, "nbformat": 4, "nbformat_minor": 4 }